Web graph compression with fast access

In recent years studying the content of the World Wide Web became a very important yet rather difficult task. Fast resource searching engines need to explore the Internet and index its whole structure while still being profitable i.e. maintaining low machine's resource consumption that implies lower costs. In order to achieve this task a new structure was introduced, a web graph. Since the Internet consists of billions of pages it is rather hard to enclose it in RAM in order to process it fast. Therefore there is a need for a compression technique that would allow a web graph representation to be put into the memory while maintaining random access time competitive to the time needed to access uncompressed web graph on a hard drive. There are already available techniques that accomplish this task, but there is still room for improvements and this thesis attempts to prove it. It includes a comparison of two methods contained in state of art of this field (BV and k 2 partitioned) to two already implemented algorithms (rewritten, however, in C++ programming language to maximize speed and resource management efficiency), which are LM and 2D, and introduces the new variant of the latter one, called 2D stripes. This thesis serves as well as a proof of concept. The final considerations show positive and negative aspects of all presented methods, expose the feasibility of the new variant as well as indicate future direction for development. wskazuje kierunek jego rozwoju. The main goal of this thesis is to elaborate on techniques of compressing web graphs that allow fast access to the successor list of a node and to show positive and negative aspects of proposed methods and other known ones as well as their practical applications. In this work three methods were evaluated and tested: LM (for list merge), 2D-stripes and 2D-nostripes. They were then compared to BV [1], the most popular compressed web graph representation in the market, and k 2 partitioned [2], which is also very promising. This thesis brings the following contributions: 1. Investigation of currently available tools and algorithms for compressing web graphs, 2. Attempt to translate LM method, which is currently available in Java, into C++ in order to check its efficiency among other C++ solutions, 3. Elaboration of demanded features of the new solution, 4. Analysis of possible advantages and drawbacks of each idea and choosing most suitable algorithms for …